Feature: Book Genie Mode - Automated Testing with Detailed Reporting

File name: BookGenie.feature

Relative path: features/BookGenie.feature

Scenarios

Chart

Status

Progress

Passed	50.00 %
Failed	50.00 %

Total duration:

00:05:39.634

Metadata

Scenario Outline: Book Genie Response Validation with Citation Verification

7
1
10
Duration: 00:00:57.904

When I open the Creative Workspace login page 00:00:03.260

Then I should see the homepage 00:00:00.042

When I click on the mode selection dropdown 00:00:03.871

Then I can see the "BookGenieQA" mode 00:00:00.023

And I select the "BookGenieQA" mode 00:00:02.566

Then I type "Suggest 5 books on christmas" on chat input element 00:00:31.236

And I wait for AI to complete thinking 00:00:08.783

And I validate the response is visible for "Suggest 5 books on christmas" 00:00:00.011

And I extract book data from BookGenie response 00:00:00.008

And I save BookGenie book data to Excel file for query "Suggest 5 books on christmas" 00:00:00.000

+ Show Error

Error: No book data extracted from BookGenie response.
    at CustomWorld.(anonymous) (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:46:15)
    at /home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:8:71
    at __awaiter (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:4:12)
    at CustomWorld.(anonymous) (/home/runner/work/HC_Epub_Automation/HC_Epub_Automation/tests/steps/bookGenie.step.ts:59:12)

And I validate each book individually against Excel file for query "Suggest 5 books on christmas" 0s

And I validate that reason texts match citation texts with 80% similarity 0s

And I generate detailed citation validation report 0s

Then I verify database connectivity 0s

And database should contain 10032 books 0s

And I validate extracted books individually against database 0s

And I validate BookGenie response relevance with AI for query "Suggest 5 books on christmas" 0s

After 00:00:08.098

+ Show Info + Show Info + Screenshot + Video

=== SCENARIO LOGS: Book Genie Response Validation with Citation Verification ===
[1:19:10 PM] Navigating to Creative Workspace login page
[1:19:14 PM] Homepage visibility: false
[1:19:14 PM] ℹ️ Clicking on mode selection dropdown
[1:19:18 PM] ✅ Mode selection dropdown clicked successfully
[1:19:18 PM] ℹ️ Checking visibility of mode: BookGenieQA
[1:19:18 PM] ✅ Mode "BookGenieQA" is visible
[1:19:18 PM] ℹ️ Selecting mode: BookGenieQA
[1:19:18 PM] ✅ Mode "BookGenieQA" selected successfully
[1:19:20 PM] ℹ️ Waiting for mode transition to complete
[1:19:20 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[1:19:50 PM] ⚠ ⚠Book Genie mode did not loaded
[1:19:50 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas"
[1:19:50 PM] ℹ️ Waiting for chat input to be visible
[1:19:50 PM] ℹ️ Clearing chat input
[1:19:50 PM] ℹ️ Typing query: "Suggest 5 books on christmas"
[1:19:51 PM] ℹ️ Pressing Enter to submit query
[1:19:51 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas"
[1:19:51 PM] ℹ️ Waiting for AI to complete thinking...
[1:19:51 PM] 📋 WAITING FOR AI RESPONSE
[1:19:51 PM] ℹ️ Looking for AI thinking indicator...
[1:19:51 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)
[1:19:53 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request
[1:19:53 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)
[1:19:57 PM] ✅ ✓ AI thinking completed - Response should be ready
[1:19:57 PM] ℹ️ 🔍 Checking for "None of the above, just" option...
[1:19:57 PM] ℹ️ ✓ "None of the above, just" option not present - proceeding normally
[1:19:57 PM] ℹ️ Allowing additional time for response rendering
[1:20:00 PM] ✅ AI response wait process completed
[1:20:00 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas"
[1:20:00 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas"
[1:20:00 PM] ℹ️ Locating the latest response in chat
[1:20:00 PM] ℹ️ Waiting for response to be visible (max 30 seconds)
[1:20:00 PM] ✅ Response received - Length: 636 characters
[1:20:00 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...That sounds like a lovely theme! To help you find...
[1:20:00 PM] ℹ️ Extracting book data from response...
[1:20:00 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE
[1:20:00 PM] ℹ️ Locating BookGenie response container
[1:20:00 PM] ℹ️ Extracting HTML content from response
[1:20:00 PM] ℹ️ HTML content extracted - Length: 9176 characters
[1:20:00 PM] ℹ️ Parsing HTML to extract book data
[1:20:00 PM] ✅ SUCCESSFULLY EXTRACTED 0 BOOKS
[1:20:00 PM] 📋 EXTRACTED BOOKS SUMMARY
[1:20:00 PM] 📊 Total books extracted: 0
[1:20:00 PM] 📊 Average relevance score: 0%
[1:20:00 PM] ✅ Extracted 0 books from response
=== END LOGS ===

Scenario Logs: Book Genie Response Validation with Citation Verification

[1:19:10 PM] Navigating to Creative Workspace login page

[1:19:14 PM] Homepage visibility: false

[1:19:14 PM] ℹ️ Clicking on mode selection dropdown

[1:19:18 PM] ✅ Mode selection dropdown clicked successfully

[1:19:18 PM] ℹ️ Checking visibility of mode: BookGenieQA

[1:19:18 PM] ✅ Mode "BookGenieQA" is visible

[1:19:18 PM] ℹ️ Selecting mode: BookGenieQA

[1:19:18 PM] ✅ Mode "BookGenieQA" selected successfully

[1:19:20 PM] ℹ️ Waiting for mode transition to complete

[1:19:20 PM] ℹ️ Typing query: "Suggest 5 books on christmas"

[1:19:50 PM] ⚠ ⚠Book Genie mode did not loaded

[1:19:50 PM] ℹ️ Preparing to type query: "Suggest 5 books on christmas"

[1:19:50 PM] ℹ️ Waiting for chat input to be visible

[1:19:50 PM] ℹ️ Clearing chat input

[1:19:50 PM] ℹ️ Typing query: "Suggest 5 books on christmas"

[1:19:51 PM] ℹ️ Pressing Enter to submit query

[1:19:51 PM] ✅ Query submitted successfully: "Suggest 5 books on christmas"

[1:19:51 PM] ℹ️ Waiting for AI to complete thinking...

[1:19:51 PM] 📋 WAITING FOR AI RESPONSE

[1:19:51 PM] ℹ️ Looking for AI thinking indicator...

[1:19:51 PM] ℹ️ Waiting for thinking indicator to appear (max 2 minutes)

[1:19:53 PM] ✅ ✓ AI thinking indicator appeared - AI is processing the request

[1:19:53 PM] ℹ️ Waiting for thinking indicator to disappear (max 5 minutes)

[1:19:57 PM] ✅ ✓ AI thinking completed - Response should be ready

[1:19:57 PM] ℹ️ 🔍 Checking for "None of the above, just" option...

[1:19:57 PM] ℹ️ ✓ "None of the above, just" option not present - proceeding normally

[1:19:57 PM] ℹ️ Allowing additional time for response rendering

[1:20:00 PM] ✅ AI response wait process completed

[1:20:00 PM] ℹ️ Validating response for query: "Suggest 5 books on christmas"

[1:20:00 PM] 📋 VALIDATING RESPONSE FOR QUERY: "Suggest 5 books on christmas"

[1:20:00 PM] ℹ️ Locating the latest response in chat

[1:20:00 PM] ℹ️ Waiting for response to be visible (max 30 seconds)

[1:20:00 PM] ✅ Response received - Length: 636 characters

[1:20:00 PM] ℹ️ Response preview: Watch Me WorkInterpreting context for your query...That sounds like a lovely theme! To help you find...

[1:20:00 PM] ℹ️ Extracting book data from response...

[1:20:00 PM] 📋 EXTRACTING BOOK DATA FROM RESPONSE

[1:20:00 PM] ℹ️ Locating BookGenie response container

[1:20:00 PM] ℹ️ Extracting HTML content from response

[1:20:00 PM] ℹ️ HTML content extracted - Length: 9176 characters

[1:20:00 PM] ℹ️ Parsing HTML to extract book data

[1:20:00 PM] ✅ SUCCESSFULLY EXTRACTED 0 BOOKS

[1:20:00 PM] 📋 EXTRACTED BOOKS SUMMARY

[1:20:00 PM] 📊 Total books extracted: 0

[1:20:00 PM] 📊 Average relevance score: 0%

[1:20:00 PM] ✅ Extracted 0 books from response

Scenario Outline: Book Genie Response Validation with Citation Verification

When I open the Creative Workspace login page 00:00:02.567

Then I should see the homepage 00:00:00.034

When I click on the mode selection dropdown 00:00:03.606

Then I can see the "BookGenieQA" mode 00:00:00.023

And I select the "BookGenieQA" mode 00:00:02.652

Then I type "Suggest 3 books for childrens" on chat input element 00:00:31.328

And I wait for AI to complete thinking 00:00:16.943

And I validate the response is visible for "Suggest 3 books for childrens" 00:00:00.010

And I extract book data from BookGenie response 00:00:00.012

And I save BookGenie book data to Excel file for query "Suggest 3 books for childrens" 00:00:00.015

And I validate each book individually against Excel file for query "Suggest 3 books for childrens" 00:00:00.018

And I validate that reason texts match citation texts with 80% similarity 00:03:24.467

+ Show Info

CITATION VALIDATION DETAILED REPORT
================================================================================

BOOK: Araminta Spookie 3: Frognapped
------------------------------------------------------------
✅ Reason 1: PASS (100%)

✅ Reason 2: PASS (95%)

❌ Reason 3: FAIL (0%)
   📝 Reason: The book is part of the "Araminta Spookie" series...
   📚 Citation: which includes the popular "Septimus Heap" series, establishing a rich context for this whimsical ta...
   ⚠️  Citation text not found in reason: "which includes the popular "Septimus Heap" series, establishing a rich context for this whimsical tale."


BOOK: A Series of Unfortunate Events #1: The Bad Beginning
------------------------------------------------------------
✅ Reason 1: PASS (100%)

✅ Reason 2: PASS (100%)

✅ Reason 3: PASS (100%)


BOOK: Replay
------------------------------------------------------------
✅ Reason 1: PASS (100%)

✅ Reason 2: PASS (100%)

✅ Reason 3: PASS (95%)


SUMMARY: 8/9 reasons passed (88.9%)
OVERALL STATUS: PASS

And I generate detailed citation validation report 00:00:00.001

+ Show Info + Show Info

Citation Validation Summary: 8/9 passed

📖 Araminta Spookie 3: Frognapped

✅ Reason 1: 100% match

✅ Reason 2: 95% match

❌ Reason 3: 0% match

⚠️ Citation text not found in reason: "which includes the popular "Septimus Heap" series, establishing a rich context for this whimsical tale."

📖 A Series of Unfortunate Events #1: The Bad Beginning

✅ Reason 1: 100% match

✅ Reason 2: 100% match

✅ Reason 3: 100% match

📖 Replay

✅ Reason 1: 100% match

✅ Reason 2: 100% match

✅ Reason 3: 95% match

📈 Validation Summary

Total Reasons Validated: 9

Passed: 8

Failed: 1

Pass Rate: 88.9%

Overall Status: PASS

Then I verify database connectivity 00:00:00.334

And database should contain 10032 books 00:00:00.247

And I validate extracted books individually against database 00:00:00.424

And I validate BookGenie response relevance with AI for query "Suggest 3 books for childrens" 00:00:06.230

+ Show Info + Show Info

PER-BOOK AI VALIDATION REPORT - Query: "Suggest 3 books for childrens"
================================================================================

OVERALL RELEVANCE SCORE: 70%
STATUS: FAIL
BOOKS ANALYZED: 3

BOOK 1: "Araminta Spookie 3: Frognapped"
------------------------------------------------------------
Overall Score: 70%

SECTION SCORES:
  Author Information: 70% - Analysis unavailable - API error
  Publishing Date: 70% - Analysis unavailable - API error
  Why Match Explanations: 70% - Analysis unavailable - API error
  Relevance Scores: 70% - Analysis unavailable - API error

DETAILED FEEDBACK:
  • AI analysis temporarily unavailable due to service overload

IMPROVEMENT SUGGESTIONS:
  • Retry analysis when service is available

================================================================================

BOOK 2: "A Series of Unfortunate Events #1: The Bad Beginning"
------------------------------------------------------------
Overall Score: 70%

SECTION SCORES:
  Author Information: 70% - Analysis unavailable - API error
  Publishing Date: 70% - Analysis unavailable - API error
  Why Match Explanations: 70% - Analysis unavailable - API error
  Relevance Scores: 70% - Analysis unavailable - API error

DETAILED FEEDBACK:
  • AI analysis temporarily unavailable due to service overload

IMPROVEMENT SUGGESTIONS:
  • Retry analysis when service is available

================================================================================

BOOK 3: "Replay"
------------------------------------------------------------
Overall Score: 70%

SECTION SCORES:
  Author Information: 70% - Analysis unavailable - API error
  Publishing Date: 70% - Analysis unavailable - API error
  Why Match Explanations: 70% - Analysis unavailable - API error
  Relevance Scores: 70% - Analysis unavailable - API error

DETAILED FEEDBACK:
  • AI analysis temporarily unavailable due to service overload

IMPROVEMENT SUGGESTIONS:
  • Retry analysis when service is available

================================================================================

SUMMARY FEEDBACK:
----------------------------------------
• AI analysis service is temporarily overloaded
• Using fallback analysis with neutral scores
• Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent: [429 Too Many Requests] You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits.
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 2
Please retry in 22.090216902s. [{"@type":"type.googleapis.com/google.rpc.QuotaFailure","violations":[{"quotaMetric":"generativelanguage.googleapis.com/generate_content_free_tier_requests","quotaId":"GenerateRequestsPerMinutePerProjectPerModel-FreeTier","quotaDimensions":{"location":"global","model":"gemini-2.5-pro"},"quotaValue":"2"}]},{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Learn more about Gemini API quotas","url":"https://ai.google.dev/gemini-api/docs/rate-limits"}]},{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"22s"}]

OVERALL IMPROVEMENT SUGGESTIONS:
----------------------------------------
• Retry the analysis when Gemini API is less busy
• Consider running tests during off-peak hours
• Check Google AI status page for service updates

Overall Relevance Score: 70%

Status: FAIL

Books Analyzed: 3